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ABSTRACT 

This policy research summary outlines the paradigms 
and processes used in recent research and evaluation studies 
conducted on supported employment and rehabilitation programs in 
Minnesota. First, the paper reviews evaluation strategies and models 
for assessing the individual and societal efficacy of supported 
employment. It identifies key effectiveness evaluation activities 
such as identifying outcomes, developing a conceptual framework, 
specifying comparisons, and specifying outcomes and indicators. 
Principles of efficiency evaluation are also addressed. Second, the 
paper presents a case study and methodology for evaluating supported 
employment costs and benefits. The case study sample included 11 
agencies at 13 sites in Minnesota, with 41 program options. Results 
are reported in terms of: consumer demographic characteristics and 
work activity, agency and program costs, and benefit-cost estimates. 
Third, the paper offers recommendations for strengthening current 
evaluation methods and practices as a means of improving the quality, 
effectiveness, and efficiency of supported employment programs. 
Recommendations focus on: the importance of reaching professional 
consensus on desired goals and outcomes; the need for improved 
conceptual frameworks; and the importance of controlling for 
individual characteristics. (Contains 44 references.) (DB) 
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This issue of Policy Research Brief summarizes the 
paradigms and processes used in several recent r esearch 
and evaluation studies conducted on supported employment 
and rehabilitation programs in Minnesota. The purpose of 
this brief is threefold. First, to provide an overview of 
evaluation strategies and models for assessing the indi- 
vidual and societal efficacy of supported employment. 
Second, to present a case study and methodology for 
evaluating supported employment costs and benefits. And, 
finaiiy, io offer recommendations for strengthening current 
evaluation methods and practices as a means of improving 
the quality, effectiveness, and efficiency of supported 
employment programs. This policy brief was prepared by 
David R. Johnson, Darrell R. Lewis, and Robert H. 
Bruininks of the Center on Residential Services and Com- 
munity Living, University of Minnesota. 



Introduction 



The recent emergence of supported employment (SE) 
programs in the United States has substantially raised 
expectations concerning the viability of employment in 
promoting the integration, productivity, and independence 
of persons with severe disabilities. SE is designed to serve 
persons who typically do not benefit from traditional time- 
limited vocational rehabilitation services. Unlike other 
vocational rehabilitation programs, such as transitional 
employment, SE is intended for persons wit'i more severe 
disabilities. Major components stressed in SE services have 
been (a) pay for rc&* work, (b) integration in the workplace 
with non-disabled co-workers, (c) long-term ongoing 
supportive services to facilitate job retention, (d) placement 
of individuals with severe handicapping conditions, and (e) 
interagency cooperation and funding of these services 
(Shafer, Wehman, Kregel, & West, 1990). 



Today, SE is being advocated on the basis of its 
positive social and economic impact on individuals and its 
economic benefits to society. The extent to which SE 
programs are fulfilling these individual and societal goals 
and outcomes remains center-stage in deliberations regard- 
ing the expansion of such services nationally. Questions 
concerning SE costs, accountability, and effectiveness are 
increasingly being asked by policymakers and professionals 
at the federal, state, and county levels. In addition to 
earnings and other related economic outcomes and benefits, 
the efficacy of SE is being judged on its capacity to achieve 
employment integration for individuals served. It may be 
argued that without a better understanding of the multi- 
dimensional nature of employment integration and its inter- 
relatedness to other SE outcomes (e.g., increased earnings 
and placement rates) and costs, the current high levels of 
public and professional support for this program alternative 
may diminish (Johnson & Lewis, in press). This will 
require that present methods for evaluating SE's efficiency 
and effectiveness (i.e, its net individual and societal worth) 
be substantially improved. 



Evaluation Strategies and Methods 



Evaluation methods and strategies vary extensively, 
depending upon the focus of evaluation (e.g., outcome 
assessments, program accountability, cost-benefits), and the 
key evaluation questions to be addressed through the 
analysis (e.g., Is SE effective in assisting persons with 
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disabilities to achieve higher levels of earnings? Overall, are 
these services cost-effective and cost-beneficial to society?). 
Evaluation schemes also vary based on the intended use of 
information and data obtained to facilitate policy develop- 
ment, program planning and development, improvements, 
cost containment, and other elements of program design and 
operations. Examining social programs from the perspec- 
tive of their effectiveness and efficiency remains the most 
central purpose of program evaluation. 

Effectiveness Evaluation 

The notion of what constitutes an "effective" SE 
program means different things to different people. Indi- 
viduals with disabilities and family members are inherently 
concerned with SE's capacity to improve individual levels 
of economic self-sufficiency and quality of life. Profession- 
als and policymakers certainly hold similar interests, but 
their attention is also directed to the broader aspects of 
program operation and management (i.e., achievement of 
agency mission, staff productivity, agency accountability, 
capacity to address differing characteristics and needs of 
individuals served, and other aspects of program delivery). 
In broadest terms, effectiveness evaluation involves 
identifying (from differing perspectives) and measuring the 
multiple goals and outcomes of a social program. 

Research and evaluation methods focusing on SE 
programs are still in their early stages of development. To 
date, evaluation studies in SE have investigated outcomes 
from a narrow and often, singular perspective, such as 
earnings and related economic outcomes, attained levels of 
physical and social integration, or improvements in quality 
of life among program participants. From a program 
effectiveness standpoint, the tendency has been to identify 
and attempt to measure an array of intangible social 
benefits, largely included within conceptual paradigms of 
inf nation and quality of life, to express the socially 
desirable benefits and outcomes (effects) of SE. While 
studies of this nature have contributed a substantial base of 
important and timely information that has aided profession- 
als and policymakers in understanding the relative merits of 
SE, rarely are these multiple economic and social outcomes 
linked into an integrated or multi-dimensional perspective. 
This has not only limited our understanding of supported 
employment's effectiveness in achieving its goals, but has 
prevented efforts to link multiple program outcomes to 
costs. Linkage is essential when attempting to derive 
meaningful cost-effectiveness measures. Such measures are 
also of critical importance in comparing alternative SE 
models and current service delivery options. 

Effectiveness evaluation commonly entails the follow- 
ing activities: (1) identifying key outcomes, goals, and 
information needs; (2) developing a conceptual framework 
to guide the evaluation process and subsequent analyses; (3) 
specifying the nature of comparisons to be made; (4) 



specifying and operationalizing multiple outcome measures 
and indicators; and (5) collecting and analyzing outcome and 
other data appropriate to addressing earlier information 
needs and questions (DeStefano, 1990; DeStefano & 
Wagner, 1990; Lewis, Johnson, Bruininks, Kallsen, & 
Guiliery, 1991; Schalock, 1988; and Schalock & Hill, 1986). 
A brief explanation of these key activities follows: 

• Identifying Outcomes, Goals and Information Needs. 
This initial step in effective evaluation involves stakehold- 
ers (i.e., individuals with disabilities, family members, 
professional staff, advisory board members) and others in 
a collaborative planning process to identify key evaluation 
questions and related outcome measures for study and 
analysis. Such participation is not only important, but also 
improves the quality of the evaluation design and support 
for results obtained later. 

• Developing a Conceptual Framework. Conceptual 
frameworks and models are often used to depict critical 
dependent and independent variables expected to influence 
outcomes in employment programs. Most models attempt 
to illustrate the interactive nature of individual and 
program variables related to SE outcomes. In simplest 
terms, a typical conceptual f\ . A^ork or model used in 
evaluating rehabilitation programs would examine the 
inter-relationships between system inputs (the client), 
intervention (rehabilitation services the client receives), 
and output/outcomes (the extent to which the client 
achieves intended employment goals). Frameworks also 
set the context for examining inter-relationships among 
key variables (or variable clusters), along with the hypoth- 
esized path of influence. Such conceptualizations are 
important in determining later methods of analysis (i.e., 
univariate or multivariate procedures). 

• Specifying Comparisons. In order to conduct program 
effectiveness evaluations, comparisons must be specified 
and available to the evaluator. Several comparisons can be 
employed in evaluating the effectiveness of SE programs: 
(a) intra-program comparisons of differing SE models (i.e., 
individual placement, enclave, mobile work crew, and 
entrepreneurial models; sheltered work vs. supported 
employment); (b) comparisons by consumer characteris- 
tics (i.e., disability type and level, gender); (c) cross- 
program/agency comparisons; and (d) longitudinally based 
comparisons of the same individuals or programs over 
time. The reader should consult other reviews describing 
detailed procedures and techniques for establishing 
comparison groups for analyses (Attkisson, Hargreaves, 
Horowitz, & Sorenson, 1978; DcS.cfano & Wagner, 1990; 
Madaus, Scriven, & Stufflebeam, 1985; Posavac & Carey, 
1980; and Rossi, Freeman, & Wright, 1975). 

• Specifying Outcomes and Indicators. Identifying, 
selecting, and operationalizing relevant and measurable 
outcomes is one of the most pressing issues in conducting 
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meaningful outcome or effectiveness evaluations of SE. 
Individual outcomes are most often categorized in 
monetary and nonmonetary terms. Monetary outcomes 
include wages received per hour or week, net annual 
earnings fless taxes paid), hours worked per week, and 
job tenure. Nonmonetary outcomes include employment 
integration, quality of life, an " oicill acquisition and 
maintenance. 

Effectiveness evaluation also assesses a program's 
accomplishments in achieving its mission and goals. 
Here, too, monetary and nonmonetary criteria and 
outcomes can be measured. Summative information can 
be developed to measure program effectiveness (e.g., 
program placement rates in community employment, 
average earnings and related work benefits attained by 
program participants, degree or level of agency change- 
over from segregated to integrated employment). 
Nonmonetary outcomes may include levels of satisfaction 
expressed by consumers, family members, and employers 
with the quality and effectiveness of employment services 
provided by the program or agency. 

Evaluations of SE programs from the broader context 
of the service delivery system and society as a whole have 
also recently been advocated (Schalock & Hill, 1986; 
Schalock, 1988; Schalock & Thornton, 1988; Wehman 
Krcgcl, & Shafer, 1989). Based on this view, networks of 
organizations dispensing a variety of social programs for 
individuals with disabilities also need clearly articulated 
goals and outcomes by which to judge the quality and 
effectiveness of their performance. 

• Analyzing Outcome Data. An extensive review of 
various approaches used in analyzing outcome data and 
information goes beyond the sc"pe of this publication. 
Decisions about how data are to be analyzed should be 
made early in the planning stages of an outcome asse? c - 
mcnt, in conjunction with decisions 3bout informativ. 
needs, variables and their measurement, data sources, and 
audiences (DeStcfano & Wagner, 1990). Detailed 
discussions of appropriate data analysis strategics and 
techniques can be found in Bolton (1987); Borg & Gall 
(1983); Klcinbaum, Kupper, & Miller (1988); Rossi ct al. 
(1975) and Walls & Tseng (1987). 

Efficiency Evaluation 

In the evaluation of programs serving persons with 
disabilities, benefit-cost analysis increasingly is being used 
to determine whether or not a particular program is "worth 
its cost" (i.e., whether the program generates outcomes for 
individuals with disabilities that justify the costs of produc- 
ing them). The federal-state vocational rehabilitation (VR) 
program was one of the first to extensively use benefit- 
methodologies in evaluating service costs and benefits 
(Bcrkowitz, 1980). Since the mid 60s, benefit-cost analysis 



also has been used in large-scale evaluations of federal 
transitional training programs for persons with disabilities 
(e.g., Kerachsky & Thornton, 1987), and in assessing other 
demonstration projects th.u have focused on the training 
and employment of individuals with disabilities (Hill & 
Wehman, 1983; Schneider, Martin, & Rusch, 1981). More 
recently, benefit-cost analysis also has been used in the 
evaluation of SE services for individuals with severe 
disabilities in comparisons with other employment and 
rehabilitative activities (e.g., Hill, Wehman, Kregel, Banks, 
& Metzler, 1987; Lam, 1986; Lewis, Johnson, Bruininks, 
Kallsen, & Guillery, 1992; Noble, Conley, Banerjee, & 
Goodman 1991). Today, almost all state vocational 
rehabilitation agencies employ some form of benefit-cost 
analysis for reporting to legislatures and policymakers on 
the likely efficiency results from traditional vocational 
rehabilitation activities. Unfortunately, most state voca- 
tional rehabilitation agencies do not systematically collect 
and report information on the costs and benefits of SE 
programs (Lewis, Johnson, Chen, & Erickson, 1992). 

The application of benefit-cost analysis as a strategy 
for evaluating employment and training programs has 
continued to be a subject of substantial controversy. This 
has been due, in large part, to insufficiencies of data 
concerning the benefits and costs of programs and to the 
extreme sensitivity of the results of benefit-cost models to 
their underlying assumptions relating to discount rates, 
earning streams, comparison groups, and the like. Several 
excellent critiques of the conceptual and methodological 
limitations of using benefit-cost analysis to evaluate 
employment and training programs for individuals with 
disabilities can be found in Conley & Noble (1990), 
Johnston (1987), Noble & Conley (1987), Rhodes, 
Ramsing,& Hill (1987), and Thornton & Maynard (1989). 

Benefit-cost analysis is essentially a comparison that 
involves several logical steps and procedures for estimating 
and valuing the full range of economic and social outcomes 
of programs. These steps include: (1) defining the program 
and its alternatives, (2) determining the analytical perspec- 
tive, (3) listing the benefits and costs, and (4) valuing 
program effects and costs. In lieu of detailing these 
procedures in this publication, a case-study addressing the 
application of a benefit-cost methodology in evaluating SE 
programs in Minnesota is offered. 



■ Case Study: A Benefit-Cost Analysis* 



In 1990, the state of Minnesota was concerned with its 
lack of knowledge about the economic outcomes of 
supported employment. Accordingly, a benefit-cost study 
was commissioned to examine this question. The study, 
initiated in 1990. focused on the development and use of a 
resource components cost model to establish benchmark 
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cost and resource use estimates for 1 1 SE service agencies 
at 13 program sites in Minnesota (Lewis, Johnson, 
Bruininks, Guillery, & Kallsen, 1991). The sample of 
agencies was drawn from five Minnesota counties and 
included five day activity centers (DACs), four vocational 
rehabilitation facilities, one mental health program, and one 
Regional Treatment Center. Data were also collected on the 
demographic characteristics and programs outcomes of 
persons with disabilities served in these agencies through 
four alternative program models: (a) habililation training, 
(b) on-site sheltered employment, (c) community group 
supported employment, and (d) community individual 
supported employment. Thus, the study sample included 1 1 
agencies at 13 sites, with 41 program options. From these 
data, several benefit-cost ratios were estimated to explain 
possible efficiency effects resulting from these programs. 

Data Sources 

The information and data on the agency resources 
employed and their respective costs were collected through 
examination of program reports, budget and audited 
expenditure records, and discussions with key service 
agency program personnel. Similarly, information and data 
on consumers and services at each of the 13 program sites 
were collected through examination of service agency and 
program records, and discussions with key service agency 
program personnel. All data on individuals with disabilities 
served through these programs were collected from agency 
files and records by agency personnel and reported in a 
manner so as to protect the confidentiality of clients. All 
resource and cost data were collected for the January 1 , 
1989, to December 31, 1989, fiscal period for six of the 
agencies, and for the July 1 , 1989, to June 30, 1990, for the 
other five agencies. All consumer and program information 
and data were collected for the same respective periods. 

Questions Posed by the Study 

The study attempted to assess four sets of questions. 
The first dealt with estimating costs associated with the 
delivery of the four types of training and employment 
services identified earlier. For each of the four sub-program 
areas, average costs were estimated per year, per day, and 
per hour on an individual client basis. A second set of 
questions dealt with estimating the benefits associated with 
the delivery of the four types of sub-programs. The benefits 
similarly were estimated in annual, daily, and hourly terms. 
A third set of questions attempted to assess the likely 
efficiency effects of the alternative service programs, and 
the results were estimated and examined in benefit-cost 

* The design and results reported for this case study are taken 
from a larger benefit-cost study of SE programs in Minnesota 
funded by the Minnesota Division of Rehabilitation Services 
(Lewis, Johnson, Bruininks, Kallsen, & Guillery, 1991 ). See 
reference, p 1 1 . 
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terms. Finally, a fourth set of questions dealt with how the 
different program costs related to differing program, client, 
and agency characteristics. Specifically, the costs of 
program operations were related to: (1) the type of training 
and employment program, (2) client characteristics, (3) the 
number of participants, (4) job placement and hours of 
work, (5) staffing ratios, and (6) geographic location of 
program, (i.e., urban vs. rural.) 

Information on individual consumers' work activities 
was collected relating to type and level of program partici- 
pation (e.g., number of daily hours and days per year within 
each sub-program area) and earnings (e.g., hourly wage 
rates and annual earnings within each sub-program area). 
Data was collected to illustrate variable rates of consumer 
participation across agencies and programs. Random 
sampling procedures were used to identify representative 
samples from the populations served by these agencies. 

Methodology 

• Cost-Accounting Framework. Tne cost-analysis 
technique employed in this study was a resource compo- 
nent approach to costing out training and human service 
programs, similar in method to several other cost studies 
undertaken by the authors (Lewis, Bruininks, & Thurlow, 
1989, 1990), and by others evaluating transitional 
employment programs (Kcrachsky et al, 1985). This 
approach requires: (1) a comprehensive listing of all the 
direct program and supplemental services within an 
agency or delivery system; (2) the identification, measure- 
ment, and valuing of the specific resources employed 
vithin each direct program and supplemental service area; 
and (3) the allocation of resource costs associated with all 
supplemental service areas to each of the direct program 
areas. On the basis of these allocated cost data and the 
number of clients, or client-days and hours of service 
provided by each program area, overall program costs 
were determined, along with various client-unit costs 
(e.g., per year, per day, or per hour). The cost analyses 
and methodology of this study focus on actual client uses 
of service (e.g., hours or days of service), not on catego- 
ries of budget or reimbursement or administrative 
classifications of clients. 

• Client Demographic Characteristics and Work 
Activities. An important aspect of the study was to 
provide a comparison of program and sub-program costs 
with descriptive information on the clients served by each 
agency within their respective programs. Although total 
costs for selected program and sub-program areas may be 
similar between agencies, when costs are adjusted for 
numbers of clients (along with variable days and hours of 
training) and differences in client demographic character- 
istics, diagnostic, and skills information, important 
differences in average unit costs can and often do occur. 
Even though analysis of agency program and sub- 
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program costs was viewed as one of the primary foci for 
the study, comparing such costs to the differing character- 
istics of clients served provided important information for 
inteipreting differing patterns of resource use. 

Two procedures were used to collect descriptive 
information on persons served by each agency across the 
distribution of sub-program areas (e.g., habitation 
training, on-site sheltered employment, group and 
individual SE, as well as competitive employment). The 
Inventory for Client and Agency Planning [ICAP] 
(Bruininks, Hill, Weatherman, & Woodcock, 1986), was 
used as the primary instrument for collecting demo- 
graphic, diagnostic, and other descriptive information on 
persons with disabilities included in the study. A second 
instrument was used to collect information on individual 
levels of work activity relating to type and level of 
program participation (e.g., number of daily hours and 
days per year by program area) and earnings (e.g., hourly 
wage rates and annual earnings by program area). These 
data were collected to illusuate the variable rates of client 
participation across agency programs. 



• Benefit Cost-Accounting Framework. An illustrative 
accounting framework employed in the study for analyz- 
ing the benefits and costs of SE appears in Table 1 . The 
perspective and taxonomy presented is based, in part, on 
previous recommendations by Thornton and Maynard 
(1989) and Noble (1977) from their earlier work in 
benefit-cost analysis of vocational rehabilitation pro- 
grams. Table 1 summarizes, as an illustration of the 
framework, the benefits and costs of SE services for 
persons with disabilities in a community program as 
compared to an on-site work setting in a day activity 
center at Agency A. All data presented in Table 1 have 
been adapted from the actual results reported in one of the 
cases (i.e., Agency A) in the study and are detailed and 
available in the Cost Benefits ofSE: Technical Appendix 
(Lewis, Johnson, Bruininks, Kallsen, & Guillery, 1991). 

The accounting framework depicts SE benefits and 
costs from the three stakeholder perspectives of consum- 
ers/families, other taxpayers, and society as a whole. The 
primary purpose of this format is to organize and conduct 
analyses so as to ensure that all major impacts of the 



■ Table 1 : Benefits and Costs of SE per Year 



Community Individual SE Versus On-Site Employment 





ANALYTICAL PERSPECTIVES 


Impacts 


Social 


Consumer 


+ Other Taxpayers 


Benefits: 








1) INCREASED PRODUCTIVITY 
Additional Earned Income 
Additional Fringe Benefits 


$2,518 
$277 


$2,518 
$277 


$0 
$0 


2) REDUCED USE OF ALTERNATIVE PROGRAMS 
Costs of On-Site Employment 


$6,345 


$0 


$6,345 


3) DECREASED GOVERNMENT SUBSIDIES 
Reductions of SSI/MA Payments 


$0 


($810) 


$810 


4) OTHER BENEFITS 

Increased Community Integration 
Increased Quality of Life 
Increased Self-Esteem 


+ 
+ 
+ 


+ 
+ 
+ 


+ 
+ 
+ 


Total Benefits: 


$9,140 


$1,985 


$7,155 


Costs: 








1 ) COSTS OF AGENCY SE PROGRAM (Individual Program) 


$3,473 


$0 


$3,473 


2) TARGETED JOB TAX CREDIT 


$108 


$0 


$108 


3) INCREASED TAXES PAID BY CONSUMER 


$0 


$353 


($353) 


Total Costs: 


$3,581 


$353 


$3,228 


Net Benefits: 


$5,559 


$1,632 


$3,927 


Benefit/Cost Ratio: 


$2.55 


$5.62 


$2.22 


Source: Table VII was adapted from Table 9 in the technical appendix of Lewis et al (1991). 

Notes: All data are reported in per consumer per year terms with 1 989 data drawn from actual consumer work experience.Net benefits would be larger if the orw 
successfully doted case in competetive employment (without support) was included in increased productivity. 
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program are captured and reported accurately and fully in 
the analysis. This type of benefit-cost framework 
provides insights not only into the monetary benefits and 
costs, but also into those effects that cannot be valued 
monetarily. Nonmonetary values include such qualitative 
benefits as increased independent living, improved quality 
of life, reduction of functional limitations, and increased 
self esteem. The framework provides a structured means 
of identifying, measuring and evaluating the full range of 
costs and benefits of SE services. Most importantly, the 
social and private perspectives are clearly visible in all 
phases of the analysis. 

Results and Findings 

• Consumer Demographic Characteristics and Work 
Activity* Table 2 reports the summary of demographics 
and work activity for the sample of participants in each of 
the 1 1 agencies and 13 program sites in the study, 
according to the four training and employment program 
options. Several generalizations can be drawn from the 
information found in this study relative to the characteris- 
tics of individuals served. First, comparisons made across 
agencies for each of the four program areas illustrate 
considerable variability in the types of individuals being 
served. As a consequence, considerable caution must be 
observed in making judgments about relative performance 
as between agencies. Second, comparisons made across 
programs indicate that level of adaptive functioning (i.e., 
average ICAP score) generally follows type of program. 
Persons with disabilities served in individual SE had the 
highest levels of adaptive functioning and the lowest 
ratings of behavior problems when compared to persons 
served in any of the other three alternative program 
models. On the other hand, within-agency comparisons 
of both adaptive and problem behaviors across program 
areas reveal few differences among individuals served in 
group SE, on-site employment, or habitation training. 
Based on trends in the data, type, level, and intensity of 
behaviors did not appear to strongly differentiate indi- 
viduals across these three program areas. 

Generally, the less severe the disability, as measured 
by both proportion of persons served with moderate to 
profound mental retardation and ICAP scores, the more 
likely the individuals were to be placed in individual SE. 
Similarly, the severer the disability, and the more inten- 
sive the services, the more likely the individuals were to 
be placed in habitation training. In these cases, the 
linkages were also reflected in generally higher average 
costs per client, and in the case of individual SE, these 
linkages were reflected in generally lower costs. How- 
ever, in the case of employment options between on-site 
employment (sheltered) and community group SE, no 
clear pattern or linkage of disability levels and costs could 
be observed either between the options or betveen 



programs within the options. 

Although that it is generally assumed that costs 
follow resource needs for support services, and that these 
services follow client needs, as reflected in the average 
client's level and type of disability, this bit of "conven- 
tional wisdom" may be overstated relative to SE. When 
combining client demographic, functional characteristics, 
and adaptive characteristics information, as reported in 
the ICAP, with the type of cost information reported in 
Table 2, average annual costs do not appear to associate 
strongly with the client's level of disability, in either on- 
site sheltered, or group SE options. Findings such as this, 
derived through evaluations of SE's efficiency effects, 
have important policy implications. Knowing, for 
example, that the placement of individuals with disabili- 
ties in on-site sheltered vs. community group SE does not 
necessarily relate to the client's level of disability or 
adaptive behaviors places into question restrictions 
programs may set on the program participation of clients. 

• Agency and Program Costs. Table 2 also reports 
selected agency characteristics and costs from the study. 
As noted in Table 2, extraordinary cost differences may 
develop in comparing alternative training and SE delivery 
systems. It is found that these cost differences result from 
a variety of factors, including a) the number of individuals 
being served, b) the community settings (i.e., employment 
opportunities nearby), c) the duration and intensity of 
services provided durii- j days and hours of training or 
work, d) the particular cr^ractcristics of the persons 
served, e) the staff-to-client ratios and staff salaries, and f) 
the type and extent of support services. 

Two general conclusions can be drawn about these 
factors. First, it is generally understood that type of 
training or employment program strongly influences the 
type and amount of resources needed to deliver services. 
For example, it appears from the results of the study that 
almost all forms of habitation training, on average, 
require almost twice as much in annual resource costs as 
do other employment options. There do net appear to be 
any consistent patterns of cost differences between type of 
SE when costs were expressed in annual terms. On the 
other hand, when costs were expressed in daily and hourly 
terms, it appeared that individual SE had the lowest costs 
among the three employment options. Second, the 
proportion of clients who are enrolled and then actually 
placed in jobs and provided support services is a key 
factor in costs. As expected, the variability between daily 
and hourly costs of the different employment options was 
largely a function of the number of days and hours in 
which the individuals were actually working. 

• Benefit-Cost Estimates- The benefit-cost accounting 
framework of this study confirms that reasonably accurate 
empirical estimates can be made with respect to assessing 
the cost-effectiveness of alternative SE programs. A 
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■ Table 2: Summary of Agency Characteristics and Costs 













MuefrativeSE 


Agenciee 


ki Minnesota (a) 












Training and Employment Options 


DAC 


REHAB 


MH 


RTC 










A 


B 


c 


D 




p 


G-1 


G-2 


G-3 




j 


K 


L 


A 


CD 


ucniAu 

MCUIAN 


Total Numb* o* Contumara 


74 


35 


72 


GO 


75 


25a 


222 


95 


134 


152 


93 


94 


528 








HabiMution Training 


































Nurrbar of consumers 


NA 


35 


63 


58 


NA 


NA 


NA 


95 


NA 


NA 


NA 


NA 


528 


155 8 


OAQ 1£ 
dKni. 10 


DO 


Average ICAP score 


NA 


64.9 


47.2 


55.8 


NA 


NA 


NA 


68.1 


NA 


NA 


NA 


NA 


46 


56 40 


10.66 


55.8 


Average ICAP maladaptive behavior score 


NA 


-18.7 


-8.1 


•11.7 


NA 


NA 


NA 


-7.7 


NA 


NA 


NA 


NA 


-17.9 


-12.82 


5.25 


-11.7 


Proportion with moderate to profound MR 


NA 


30% 


78% 


66% 


NA 


NA 


NA 


77% 


NA 


NA 


NA 


NA 


76% 


66% 


20% 


76% 


Average hours per day 


NA 


2.89 


3.42 


5.12 


NA 


NA 


NA 


2 83 


NA 


NA 


NA 


NA 


3.94 


3.64 


0.94 


3.42 


Average annual costs per consumer 


NA 


$6 380 


$8 017 


$7697 


NA 


NA 


Tin 


$4,222 


U A 

Tin 


U A 

Tin 


NA 


NA 


$10,698 


$7,403 


$2,370 


$7,697 


Average dairy costs per consumer 


NA 


$42 98 


$45 17 


$34 12 


NA 


klA 

Tin 


Tin 


$21 .21 


NA 


NA 


NA 


NA 


$54.98 


$39.69 


$12.72 


$42.96 


Average hourly costs per consumer in attendance 


NA 


$14.87 


$1321 


$6.66 


NA 


NA 


NA 




NA 


NA 


PJ A 

Tin 


NA 


$13.95 


$1 1 .24 


$3.85 


$13.21 
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Nurrber of consumers 


11 


26 


34 


49 


NA 


240 


216 


95 


95 


120 


77 


NA 


316 


116 


99 06 


96 


Average [CAP score 


76 


63.7 


43.8 


59.1 


NA 


73.5 


83.3 


69 


77.6 


75.5 


80 9 


NA 


49.1 


68 32 


12 96 


73 5 


Average ICAP maladaptive behavior score 


■8.2 


-19.2 


-8.7 


-11.7 


NA 


-9 


-3.5 


-7.3 


-7.1 


-7.6 


-4.3 


NA 


-18.5 


-9 55 


5 10 


-8 2 


Proportion with moderate to profound MR 


20% 


25% 


79% 


66% 


NA 


14% 


21% 


76% 


23% 


34% 


54% 


NA 


74% 


43% 


26% 


34% 


Average hours per day 


6 59 


0 30 


n aa 


C.C-i 


Tin 


5.17 


5.61 


2.63 


5.37 


6.52 


4.97 


NA 


2.01 


3.84 


2.28 


4.97 


Average annual costs per consumer 


$6 345 


$1 ,830 




$1 ,930 


NA 


$5,990 


$5,636 


$4,300 


$1,649 


$9,593 


$8,616 


NA 


$2,886 


$4,517 


$2,952 


$4,300 


Average daHy costs per consumer 


tin aq 


$28.49 


$25.75 




NA 


$35.99 


$33.03 


$21 .83 


$56.73 


$51.57 


$44.93 


NA 


$16.47 


$33.08 


$14 77 


$33.03 


Average hourly costs per consumer hour employee 


$6 14 


$94 97 




$3 86 


NA 


$6.96 


AQ 


$8.30 


$10.56 


$7.91 


$9.04 


NA 


$8.19 


$17.43 


$26.65 


$8.19 


Group Supported Employment 


































Number of consumers 


52 


22 


51 


34 


32 


40 


49 


18 


g 


35 


82 


NA 


41 


38.50 


19.48 


40 


Average ICAP score 


78.2 


62.4 


55.1 


65.1 


67.3 


80 6 


82 8 


65 4 


77 7 


75 8 


80 7 


NA 


64.5 


71 .30 


9.01 


75.8 


Avorage [CAP maladaptive behavior score 


-6 


-20.8 


-7.7 


-8.1 


-18.3 


■9.3 


-3.4 


-8.6 


-6 8 


•8 


-4 1 


NA 


-16 2 


-9.78 


5.59 


-8.05 


Proportion with moderate to profound MR 


34% 


29% 


73% 


62% 


16% 


12% 


21% 


88% 


13% 


32% 


54% 


NA 


58% 


41% 


25% 


34% 


Average hours per day 


6.12 


0.52 


2.42 


2.48 


6.05 


3.90 


5.47 


4.94 


4.30 


5.53 


363 


NA 


2.80 


4.01 


1.72 


4.30 


Average annual costs per consumer 


$6,371 


$4,358 


$3,344 


$2,870 


$5,606 


$4,670 


$1,993 


$1,794 


$1,385 


$4,184 


$2,260 


NA 


$6436 


$3,689 


$1,656 


$3,764 


Average datty costs per consumer 


(30.99 


$90.80 


$35.38 


$30 63 


$4356 


$36.40 


$74.56 


$43.58 


$28.47 


$39.25 


$21 .08 


NA 


$38.62 


$41.94 


$20.76 


$37.51 


Average hourly costs per consumer hour employee 


$5.06 


$174.62 


$14 62 


$8.32 


$7.20 


$9.33 


$13.63 


$882 


$6.62 


$7.10 


$5.81 


NA 


$13.79 


$22.91 


$47.69 


$857 


Individual Supported Employment 


































Nurrtoer of consumers 


17 


19 


11 


2 


45 


27 


81 


30 


70 


48 


8 


94 


2 


34.92 


30.61 


27 


Average ICAP score 


80.7 


72.8 


74.3 


775 


76.3 


84.6 


85.8 


664 


79 9 


80 


83 6 


92.4 


74 


79.10 


6.69 


79.9 


Average ICAP maladaptive behavior score 


-6.7 


-14 5 


■1.8 


-13 


-11.6 


-7.4 


-3 


-6.4 


-7.8 


-6.6 


-4 3 


-4.4 


-6 


-7.19 


3.79 


-6.6 


Proportion with moderate to profound MR 


24% 


16% 


64% 


0% 


9% 


10% 


18% 


67% 


14% 


30% 


50% 


0% 


100% 


31% 


30% 


18% 


Average hours per day 


4.79 


1.89 


4.32 


1.00 


4.83 


4.59 


4.88 


4.20 


5.93 


5.67 


7.38 


7.75 


5.00 


4.79 


1.85 


4.83 


Average annual costs per consumer 


$3,473 


$3,580 


$4,009 


$4,193 


$1,549 


$5,437 


$2,024 


$2,035 


$2,029 


$4,571 


$451 


$958 


$5,168 


$3,037 


$1,625 


$3,473 


Average daily costs per consumer 


$17.92 


$32.07 


$26.42 


$18.39 


$11.66 


$43.34 


$16.95 


$12.86 


$13 53 


$33.39 


$2 00 


$15.54 


$21.53 


$2043 


$10 93 


$17.92 


Average hourfy costs per consumer hour employed $3.74 

1 


$16.97 


$6.12 


$18.39 


$241 


$9.44 


$3.47 


$3.06 


$2.28 


$5.89 


$0.27 


$2.01 


$4 31 


$6.03 


$5.66 


$3.74 



Note: (a) Each agency Is Identified in the original study by Lewis et al. (1991). 

NA = Program option not available in agency. X = Average of aN programs. SD = Standard Deviation. 



summary of the benefit-cost ratios is estimated for each of 
the agencies and their alternative programs in the study, 
and presented in Table 3. These ratios are presented in 
terms of both average and hourly data, and, as noted 
earlier, provide measures of the monetary efficiency of 
alternative employment options. The results indicate that 
all forms of SE are cost-effective when compared with 
habilitation training, with annual returns for society 
averaging over $2 for each $1 invested in SE. When SE 
programs (i.e., both group and individual) are compared 
with on-site employment, as noted in Table 3, positive 
results for society are indicated in seven of the 1 1 
agencies and in 15 of the 22 SE programs, with annual 



returns for most of the agencies as a group averaging 
between $1.30 and $4.00 for each $1 invested. 
Among the four options examined in the study, the 
clearest case for monetary efficiency from the societal 
perspective is found when individual SE options are 
compared with all other training and employment options. 
In a clear majority of comparative cases (i.e., 23 of 28 
cases), individual SE programs were found to have 
positive benefit-cost ratios greater than one. These results 
come about largely through the higher wage rates, longer 
hours of work, and generally lower service costs found 
within individual SE programs. 

Beyond the potential financial savings to society 
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SOCIAL COSTS AND BENEFITS 
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resulting from most of these programs, it can be antici- 
pated that the amount of increased disposable income 
available to individuals with disabilities participating in 
these programs is likely to increase their opportunities for 
greater community participation and social integration. It 
is clear that all of the nonmonetary benefits (such as 
increased community integration, quality of life, and self 
esteem) accruing to both individuals and society as a 
consequence of SE are an addition to whatever monetary 
effects may result 



■ Future Considerations in SE Evaluation 



Reaching Consensus on Goals and Outcomes 

There remains a lack of professional consensus on the 
goals and outcomes of SE for individuals with disabilities. 
The question of whether to evaluate SE solely on the basis 
of economic criteria (i.e., cost-efficiency, cost-benefit, or 
economic impact on individuals' earnings) or to focus on 
the social benefits to participants (i.e., social integration, 
quality of life, increases in skill levels) continues with 
considerable debate among researchers and professionals. 
Increasingly, researchers (e.g., Parent, Krcgel, Twardzik, & 
Metzler, 1990; Rusch, Chadsey-Rusch, & Johnson, 1991) 
have even argued that valuing SE strictly on the basis of its 
cost-effectiveness is illegitimate, and that SE should be 
valued primarily from its capacity to assist individuals in 
achieving social and community integration. Agreement on 
critical dimensions and outcome-indicators to measure 
social and community integration, quality of life, and other 
qualitative outcomes of SE, however, has not been achieved. 
Consensus on common quantitative and qualitative outcome 
measures is of critical importance when attempting to 
evaluate the effectiveness and efficiency of programs across 
differing settings and contexts. 

Improved Conceptual Frameworks 

Improved conceptual and analytic frameworks and 
methods are critically needed to understand the full impact 
of SE programs. Recently, there have been a number of 
investigations that have sought to improve upon the mea- 
surement of community adjustment through the develop- 
ment and validation of multi-dimensional outcome mea- 
sures. This research has, in part, investigated community 
adjustment as a function of several dimensions including 
demographics; employment activities; education, employ- 
ment, job training, or day habitation arrangements; family 
and friend social network; community involvement; 
personal satisfaction level; and financial independence (sec 
Bruininks, McGrew,Thurlow,& Lewis, 1990; Halpern, 
Nave, Close, & Nelson, 1986; McGrew, Johnson, & 



Bruininks, 1992). A variety of factor analytic and other 
multivariate procedures to derive composite variables were 
employed in these studies. 

The construction of composite variables based on the 
reduction of many single outcome variables aids researchers 
and evaluators in two ways. First, the sheer number of 
variables included in outcome studies of employment and 
community services can be reduced. This not only saves 
valuable time during data collection and analysis, but 
reduces the tendency to misinterpret results when inter- 
relationships among many variables must be explained. 
Second, validation of multi-dimensional outcome measures 
may contribute to the development and empirical evaluation 
of comprehensive models of employment integration and 
related dimensions of community adjustment for individuals 
with severe disabilities. Considerable experimentation and 
research must ensue in coming years to fully capture the 
complex, multi-faceted nature of SE programs. 

Controlling for individual Characteristics 

In order to provide for meaningful cross-program and 
cross-agency comparisons of the impacts and benefits of SE 
services on individuals with disabilities, standard measures 
for describing individuals are needed. In the case-study 
described earlier (Lewis, Johnson, Bruininks, Kallsen, & 
Guillery, 1991), the Inventory for Client and Agency 
Planning [ICAP] (Bruininks, Hill, Weatherman, & Wood- 
cock, 1986), was used as the statistical control of individual 
characteristics for examining the relative costs and impacts 
of supported employment The ICAP yields standard 
scores, which are useful for aggregating and analyzing 
information across individuals and programs. Most other 
evaluation studies of SE report only IQ scores or generalize 
to broad disability classification schemes. These indices 
provide only a limited understanding of the relationship of 
individual attributes and characteristics to SE outcomes and 
costs. Standardized instruments such as the ICAP aid the 
evaluator in making comparisons of program outcomes and 
costs on a wider range of individual characteristics (i.e., 
diagnostic status, functional limitations, adaptive behavior, 
problem behavior, as well as the service-related status, and 
other needs of individuals with disabilities). This is 
important in broadening our understanding of the impact of 
SE on individuals and facilitating improvements in cross- 
agency and cross-program comparisons. 

• Establishing Common Accounting Frameworks. SE 

agencies need to attend to developing a common account- 
ing framework for the reporting of their costs. Without 
such a common framework, only limited progress is likely 
in achieving more effective or efficient deployment of 
resources in SE and for making cost and outcome 
comparisons over time or between programs. The 
ultimate benefit of conducting cost and outcome analyses 
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of such service programs is to create greater understand- 
ing of the use of public resources, and the development of 
strategies to achieve the most appropriate and effective 
service programs in the most efficient manner. The 
principal value of the resource components paradigm for 
estimating costs illustrated in the case-study presented 
earlier (Lewis, Johnson, Bruininks, Kallsen, & Guillery, 
1991) lies in its comprehensiveness and accuracy. With 
an appropriately framed cost model, it is feasible for 
administrators and policymakers to evaluate resource 
usage in current programs and to stimulate the resource 
consequences of different policy and administrative 
alternatives. Similarly, with an appropriately framed 
benefit-cost accounting framework, it is possible to 
examine questions of cost-effectiveness, both from the 
perspective of society and of the individuals served. 

• Understanding the Cost-Effectiveness of the Sup- 
ported Employment Programs. Little attention has 
been directed to evaluating the cost-effectiveness of SE 
programs. For both public policy development and 
individual level program planning, information about the 
relative cost-effectiveness of alternative programs for 
persons with disabilities is critically needed. Attempting 
to find relationships between program costs and effective- 
ness measures in SE has been difficult. Nevertheless, 
program alternatives sharing similar goals can be com- 
pared according to their relative costs and outcomes. 
Those alternatives with the lowest relative cost effective- 
ness ratios could be considered the most promising with 
respect to use of society's resources, uses the fewest 
resources to achieve the program's goals. 

When the evaluation context is one in which multiple 
criteria and attributes must be considered, as in SE, the 
challenge to the evaluator or choice maker is how to 
convert these multiple outcomes into a single index that 
can be compared across several alternatives or options 
(Lewis, Erickson, Johnson, Bruininks, 1991). Determin- 
ing appropriate and meaningful indices against which the 
relative cost-effectiveness of SE can be evaluated and 
compared with other service delivery alternatives (e.g., 
sheltered employment, work activity, day activity 
programs) is a highly challenging task. Further, issues 
related to the cost-effectiveness of using alternative 
models of SE (e.g., individual placement, enclave, 
mobile, work-crew, or entrepreneurial models), and 
whether or not individual or group placement models are 
more cost-effective remain unresolved. Such information 
is, however, important from the standpoint of both pubKc 
policy and program management, as efforts to expand 
employment services continue nationally. 

Here the critical importance of achieving consensus 
on the multi-dimensional nature of SE outcomes is 
evident. If community integration, for example, is valued 
and viewed as the single most important outcome and 



attribute of SE for individuals with severe disabilities, 
more sophisticated methods for measuring and communi- 
cating this important dimension will be needed. Thus far, 
this valued dimension has not adequately been accounted 
for in evaluations of program costs and outcomes. 
Researchers must continue to search for ways to formu- 
late valid and reliable composite outcome indicators for 
use in conducting meaningful cost-effectiveness evalua- 
tions of SE programs. 



■ Conclusion 



Center-stage in SE's short history has been the need for 
ongoing evaluations of its social and economic benefits and 
outcomes. Reliable and complete outcome and cost 
information is fast becoming an essential aspect of federal, 
state, and local decision-making, and a necessity for 
planning and improving SE programs and services. The 
evaluation models and strategies described in this publica- 
tion address the difficulties and complexities involved in 
evaluating the efficiency and effectiveness of SE programs. 
Researchers and policymakers are encouraged to experiment 
with their applicability and relevance in future evaluations 
of SE services. 



■ References 



• Attkir on,C. C, Hargrcaves, W. A., Horowitz, M. J., & Sorcnson, 
L.E. (Eds ).(1978). Evaluation of human services programs . New 
York: Academic Press. 

• IWkowitz. R. D. (1980^. Rehabilitation: The federal government's 
response to disability. 1935-1954 . New York: Arno Press. 

• BerkowitzM.,&Bc! , .vOwitz,E.D.(1983). Rehabilitation research 
review: Benefit-cost analysis . Washington DC: LlS. Department 
of Education, the National Rehabilitation Information Center. 

• Bolton, B., (1987). Scores and norms. In Brian Bolton (Ed.), 
Handbook of measurement and evaluation in rehabilitation (2nd 
ed) (pp. 3-20). Baltimore: Paul H. Brookes Publishing Co. 

• Borg, W. R., & Gall, M. D. (1983). Educational research: An 
introduction . New York: Longman. 

• Bruininks, R. H., Hill, B. K., Weatherman, R. F., & Woodcock, R. 
W. (1986). Inventory for client and agency planning . Allen TX: 
DLM Teaching Resources. 

• Bruininks, R. H., McGrcw, K., Thurlow, M., & Lewi D. R. 
(1990). Dimensions of community adjustment among young 
adults with intellectual disabilities. In W. Frascr (Ed.), Key issue:, 
in mental retardation research , (pp. 435448). London: Routlcdgc. 

• Conlcy, R. W., & Noble, J. H. (1990). Benefit-cost analysis of 
supported employment. In F. R. Rusch (Ed.), Supported cmploy- 



ERLC 



11 



mcnt: Models, methods, and issues (p. 271-287). Sycamore IL: 
Sycamore Publishing Co. 

• DeStcfano, L. (1990). Designing and implementing program 
evaluation. In F. R. Rusch (Ed.), Su pported employment: Models, 
methods, and issues . DcKalb IL: Sycamore Publishing Co. 

• DeStefano. L.. & Wagner. M. (1990). Outcome assessment in 
special education: Lessons learned . Rockville MD: WESTAT. 

• Halpem. A. S., Nave, G., Close. C, & Nelson. D. (1986). An 
empirical analysis of the dimensions of community adjustment for 
adults with mental retardation in semi-independent living pro- 
grams. Australia and New Zealand Journal of Developmental 
Disabilities . 12(3). 147-157. 

• Hill, M.. & Wchman, P. (1983). Cost benefit analysis of placing 
moderately and severely handicapped individuals into competitive 
employment Journal of the Association for the Severely Handi- 
capped. 8. 30-39. 

• Hill. M.. Wchman. P., Krcgcl. J., Banks, P. D.. & Mctzler, H. M. 
(1987). Employment outcomes for people with moderate and 
severe disabilities: An eight-year longitudinal analysis of sup- 
ported competitive employment. Journal of the Association for the 
Severely Handicapped ]2, 182-189. 

• Johnson. D. R.. & Lewis, D. (in press). Supported employment: 
Program models, strategics and evaluation perspectives. In M. 
Hayden & B. Abery (Eds.), Challenges for a service system in 
transition: Ensuring quality experiences for people with develop- 
mental disabilities . Baltimore: Paul H. Brookes Publishing Co. 

• Johnston, M. V. (1987). Cost-benefit methodologies in rehabilita- 
tion. In R. Fuhrcr (Ed), Rehabilitation outcomes: Analysis and 
measurement (pp. 99-1 13). Baltimore: Paul H. Brookes Publish. 

• Ke.achsky, S.,& Thornton, C. (1987). Findings from the STETS 
transitional employment demonstration. Exceptional Children . 
53(6), 515-521. 

• Kcrachsky, S., Thornton, C, Bloomcnthal, A., Maynard, R., & 
Stephens, S. (1985). Impacts of transitional employment on 
mentally retarded young adul's: Results of the STETS demonstra- 
tion . Princeton NJ: Mathcmatica Policy Research. 

• Klcinbaum, D. G., Kupper. L. L.. & Miller, K. E. (1988). Applied 
regression analysis and other multi -variable methods . Boston: 
PWS-KENT Publishing Co. 

• Lam. C. S. (1986). Comparison of sheltered and supported work 
programs: A pilot study. Rchabil itation Counseling B ullctiru 30(2). 
66-82. 

• L/;w?'. O.. Bruininks. R.. &Thurlow, M. (1989). Cost analysis for 
/istrici tt vel special education planning, budgeting and adminis- 

2 ung. ;•. aimal of Education Finance . 14(4), 466-483. 

• Lewis. D. Bruininks, R., &Thurlow, M. (1990). Benchmark cost 
analysis of special schools for students with mental retardation. 
Journal of Special Edu cation. 24(1). 33-50. 

• Lewis. D. R., Erkkson, R. N., Johnson, D. R., & Bruininks, R. H. 
(1991). Using Multiattributc utility evaluation techniques in spe- 
cial educatio n. St. Paul: Minnesota Department of Education, 
Unique Learner Needs Section. 



• Lewis, D. R., Johnson, D. R., Bruininks, R. H. & Kallsen, L. A. 
(1993). Accounting for costs in habitation training, sheltered 
workshops, and supported employment. Education and Training in 
Mental Retardation. 28(1), 75-89. 

• Lewis, D. R., Johnson, D. R., Bruininks, R. H. Kallsen. L. A.. & 
Guillcry. R. P. (1991). Costs and benefits of supported employ- 
ment in Minnesota. Minneapolis: Institute on Community Integra- 
tion (U AP), College of Education, University of Minnesota. 

• Lewis, D. R., Johnson, D. R., Bruininks, R. H.. Kallsen, L., & 
Guillery, P. (1992). Is supported employment cost-effective in 
Minnesota? Journal of Disability Policy Studies. 3(1), 67-92. 

• Lewis, D. R., Johnson, D. R., Chcn,T. H., & Erickson, R.N. (1 992). 
On the use and reporting of benefit-cost analyses by state voca- 
tional rehabilitation agencies. Evaluation Review. 16(3), 266-287. 

• Madaus, G.. Scriven, M. & Stufflcbeam, D. (1985). Evaluation 
models: Viewpoints on educational and human services evalua- 
tion . Boston: Kluwer-Nijhoff Publishing. 

• McGrcw, K., Johnson, D. R., & Bruininks, R. H. (1992). Factor 
analysis of community adjustment outcome measures for young 
adults with mild to severe disabiliues . Manuscript submitted for 
publication. 

• Ncble, J. H., Jr. (1977). The limits of cost- benefit analysis as a 
guide to priority setting in rehabilitaUon. Evaluation Quarterly . 
1(3), 347-379. 

• Noble, J. H., Jr., & Conlcy, R. W. (1987). Accumulating evidence 
on the benefits and costs of supported and transitional employment 
for persons with severe disabiliues. Journal of The Association of 
Persons with Severe Handicaps . 12(3). 163-174. 

• Noble. J. H., Conlcy, R. W., Bancrjcc, S., & Goodman, S. (1991). 
Supported employment in New York State: A comparison of 
benefits and costs. Journal of Disability Policy Studies. 20 )* 
39-7. 

• Parent, W. S., Krcgcl, J.,Twardzik, G., & Mctzler, H. M. D. (1990). 
Social integration in the workplace: An analysis of the integration 
activities of workers with mental retardation and their coworkers. 
In J. Kregcl, P. Wchman, & M. Shafcr (Eds.), Supported employ- 
ment for persons with severe disabilities: From research topractice. 
Volume HI . Richmond: Virginia Commonwealth University Re- 
habilitation Research and Training Center. 

• Posavac,E. J.&Carcy,R.G.(1980) Program evaluation: Methods 
and case studies . Engclwood Cliffs NJ: Prentice Hall. 

• Rhodes, L., Ramsing, K., & Hill, M. (1987). Economic evaluation 
of employment services: A review of applications. Journal of the 
Association for Persons with Severe Handicaps . 12 . 175-181. 

• Rossi, P. H., Freeman, H. E., & Wright, S. R. (1975). Evaluation: 
A systematic approach . Beverly Hills CA: Sage Publications. 

• Rusch, F. R., Chadscy-Rusch, J., & Johnson, J. R. (1991). Sup- 
ported employment: Emerging opportunities for employment inte- 
gration. In L. Meyer, C. Peck, & L. Brown (Eds.), Critical issues 
in the lives of people with severe disabilities (pp. 145-169). 
Baltimore: Paul H. Brookes. 



13 



12 



Schalock, R. L. (1988). Critical performance evaluation indicators 
in supported employment- In P. Wehman & M. S. Moon (Eds.), 
Vocational rehabilitation and supported employment (pp. 163-174). 
Baltimore: Paul H. Brookes. 

Schalock, R.L.,&Hill t (1986). Evaluating employment services. In 
W. K. Kiernan & J. A. Stark (Eds.), Pathways to employment for 
adults with developmental disabilities (pp. 285-302). Baltimore: 
Paul H. Brookes. 

Schalock,R.L.,&Thornton,C(1988). Program evaluation: A field 
guide for program administrators . New York: Plenum. 

Schneider, K„ Martin, J M & Rusch, F. (198 1). Cost versus benefits 
between sheltered and nonsheltered vocational training programs: 
Are we sacrificing quality? Counterpoint. (November) 1-28. 

Shafer, M.S., Wehman, P., Kiegel, L, & West, M. (1990). National 
supported employment initiative: A preliminary analysis. American 
Journal of Mental Retardation. 25(3), 316-327. 

Thornton, C, & Maynard, R. (1989). The economics of transitional 
employment and supported employment. In M. Berkowitz & M. A. 
Hill (Eds.). Disability and the labor market . Ithaca: Cornell Univer- 
sity, New York State School of Industrial and Labor Relations; ILR 
Press. 

• Walls, R. T., & Tseng, M. S. (1987). Measurement of client 
outcomes in rehabilitation. In Bolton (Ed.), Handbook of measure- 
ment and evaluation in rehabilitation (2nd ed) (pp. 1 83-201. Balti- 
more: Paul H. Brookes Publishing Co. 

> Wehman, P., Kregel, J., & Shafer, M. S. (1989). Emerging trends in 
the national supported employment initiative: A preliminary analy- 
sis of twenty-seven states . Richmond VA: Virginia Commonwealth 
University, Rehabilitation Research and Training Center. 



Policy Research Brief 
May 1993* Vol. 5, No. 2 

Policy Research Brief is published by the Research 
and Training Center on Residential Services and Commu- 
nity Living, Institute on Community Integration (UAP), 
College of Education, University of Minnesota. This 
publication is supported, in part, by Cooperative Agree- 
ment #H133B80048 from the National Institute on 
Disability and Rehabilitation Research (NIDRR). The data 
reported were collected with support from the Minnesota 
Department of Rehabilitative Services DRS Grant* 
07DD0282101. 

This issue is authored by David R. Johnson, Darrell R. 
Lewis, and Robert H. Bruininks. issue editor is Mary 
Hayden. Managing editor is Vicki Gaylord. The opinions 
expressed are those of the authors and do not necessarily 
reflect the position of NIDRR or DRS. 

Additional copies of Policy Research Brief are 
available for $1 each. To order, send check or purchase 
order payable to University of Minnesota to the Institute 
on Community Integration, University of Minnesota, 
109 Pattee Hall, 150 Pillsbury Drive SE, Minneapolis, 
MN 55455 • (612) 624-4512. Policy Research Brief is 
available in alternative formats upon request; to request an 
alternative format contact the publications office at the 
above address/phone. 

The University of Minnesota is an equal opportunity 
employer and educator. 



Center on Residential Services and Community Living 

institute on Community Integration 

University of Minnesota 

109 Pattee Hall, 150 Pillsbury Drive SE 

Minneapolis, MN 55455 



Address Correction Requested 



Non Profit Org 
U.S. Postage 
PAID 
Minneapolis, MN 
Permit No. 155 



EPIC 33> 
CLEAR 'GHS HANDICAPPED 'GIF T 
1920 ASSOCIATION DP: 
RES ION VA 22051 



ERLC 



14 



